Codecov Report

❌ Patch coverage is 42.30769% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.43%. Comparing base (f09f8af) to head (6e124b4).
⚠️ Report is 449 commits behind head on main.

@@             Coverage Diff              @@
##               main    #2273      +/-   ##
============================================
+ Coverage     56.12%   57.43%   +1.30%     
- Complexity      976     1287     +311     
============================================
  Files           119      146      +27     
  Lines         11743    13387    +1644     
  Branches       2251     2377     +126     
============================================
+ Hits           6591     7689    +1098     
- Misses         4012     4431     +419     
- Partials       1140     1267     +127     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

That looks incredibly useful!

I have test failures with columnar shuffle trying to call unimplemented method getLong on CometListVector. I suspect that this is related to writing shuffle output from a partial aggregate. I am putting this on hold for now and perhaps try and get this into the 0.11.0 release

I understand the issue now. For COUNT(DISTINCT bool_col) the partial count outputs all of the distinct values for bool_col so we have a boolean list vector containing [[true], [false]]. Columnar shuffle writer does not understand this and thinks that the schema is a single LongType representing the final output from the count.

I understand the issue now. For COUNT(DISTINCT bool_col) the partial count outputs all of the distinct values for bool_col so we have a boolean list vector containing [[true], [false]]. Columnar shuffle writer does not understand this and thinks that the schema is a single LongType representing the final output from the count.

If I fall back to Spark for the shuffle, I see the same issue. So the problem is perhaps that we report the wrong schema as the output from the partial aggregate

I learned a lot from this WIP PR and have documented my findings in #2292 so I will close this for now.